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(54) Indexing of data sets. 



(57) A system is disclosed for facflitating later 
access to a portion of a datastream for retrieval 
purposes based upon identifying "episodes" in 
data from other sources related to a user's 
activities. The stored data, in raw or "episodic " 
form, are accessed as desired using parameters 
based on recollections of the user's or database 
interrogator's activities at the time of the event 
which the user or interrogator wishes to access 
or display. Such other sources may include 
information about the movements, gatherings, 
conversations, phone calls, etc. of users in a 
particular environment fitted with the system. 
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This invention relates to the activity-based index- 
ing of data sets, in order to facilitate the fast retrieval 
at a later date of data of interest The invention is - 
based on there being at least one set of data defining 
activity. The retrieval process involves identifying an 5 
index point in a data set. Where there are multiple 
data sets, having index points in one of them facili- 
tates access to the other set to reach the data in it at 
the point in time, or period, contemporaneous with the 
id ntified index point ■ 10 

The invention focuses upon improving access to 
information that humans find particularly difficult to 
retrieve. It exploits a suggested autobiographical 
m del of human memory which observes that humans ... 
seem to organise their memory in episodes. . js 
Associated with each episode are the properties: 
Where? When? With whom? and, In pursuit of what \ 
goal? an episode occurred. Humans seem able to 
recall these key properties of an episode long after the , 
d tails of the episode itself have been forgotten. The 20 
present invention is concerned with building a system 
that enables retrieval of episodes that would other- 
wise be difficult or impossible to retrieve, including 
matters for which records are not normally kept This 
system has been called a 'Human memory pros- 25 
th sis'. . . . v.... ... 

The present invention relies on a variety of . .. 
apparatuses for monitoring, recording and. time- 
stamping various key aspects of human activity, from - , 
which a readable list of episodes, or 'activity diary', 30 
can be generated automatically. This diary, which cor- 
relates quite closely with human, recollections of past ■ > 
episodes, can be used simply to stimulate further - . 
r call, or to identify an episode and its time of occurr-, 
ence, or to show a set of episodes from different sets: . 35 
The times may be used to locate the corresponding v 
items in any other set of time-stamped data, such as 
a sequence of video frames or an audio record of a . 
meeting. "... . . - .. 

In what follows, an 'episode' is defined as - a 40 
period of time containing a memorable sequence of 
events -. By dividing a long series of events into such c 
episodes, sequences are formed that are particularly : 
m morabie. Episodes have flags associated with 
th m that characterise them and can be used for ret- 45 
rieval. r 

One form of such apparatus was disclosed in an^ 
article in The Sunday Times , 1 1 February 1990, enti- . 
tied "Computers acquire social skills", by Jane Bird. 
This mentioned the use of identity buttons to enable so 
the computer to know who was where in the building 
at all times. Associated with the system is a network 
of microphones and/or video cameras capable of 
recording conversations. It gave the example of a 
copier in a room monitored by the computer. Upon th 55 
copier's devel ping a fault a frustrat d would-be user 
tries to recall what a colleague told him about how to 
cur the fault Th computer is interrogated to find th 



last time that the two people concern d were at the 
copier. Once this particular bit of information had been 
recovered, it was used to access the time-stamped 
audio records of what was said in the room containing 
the copier at that point of time. In this way, the enquirer 
had the opportunity of hearing again the conversation 
between him and his colleague. A system of the 
foregoing type also is described in a copending U.S. 
Patent Application of the instant inventors, which was 
filed 7 February 1991 under Serial No. 07/652, 159 on 
"Indexing of Audio/Visual Data". 

The aforementioned system has several practical 
limitations. These include the need on the part of the 
user to devote a lot of effort to identifying a particular 
time and place from the raw identity button data, and 
the lack of precise episode definition from such data. 
Also the system captures just the location of users, 
and not their activity or other significant episodes. 
These drawbacks would make the system of relatively 
little practical value. 

Another relevant publication in this area of 
technology is the article entitled "Reflexive CSCW; 
supporting long-term personal work" by H Thimbleby, 
S Anderson and I Witten, Interacting with Computers , 
volume 2, No. .3, December 1990, pages 330-336. 
The term "CSCW" means - computer-supported 
cooperative work. The section under the heading 
"Background ", on pages 331/2 of the article, refers to 
a method whereby a research worker is able to be 
given on request a record of workstation activities he 
was undertaking, over a chosenperiod. The article 
suggests that this approach can generate useful infor- 
mation for retrieval purposes, but offers no practical 
means of doing so. Like the first mentioned system , 
it fails to recognise the need for analysis of the raw 
data to produce user-recognisable episodes. 

The present invention is based on devising pat- 
tern-matching techniques for constructing recognis- 
able episodes from raw data. In this way it offers the 
user the means to arrive quickly at a particular point 
in one dataset, possibly in order to be able to obtain 
access more efficiently and quickly into at least one 
other dataset This speed of access is one of the 
essential requirements in making the technique of 
practical value. 

The invention also takes into account situations 
where there may be relatively infrequent change of 
location by users, i.e. where location data would be of 
little use in identifying episodes with precision. In 
these situations, other forms of activity are likely to 
provide a more effective means of episode identifi- 
cation. The invention therefore presents means of 
instrumenting these activities and analysing the raw 
data, thus constructing recognisable pisodes from 
these multipl sources of activity data. The inv ntion 
also obs rves that retri val often involves correlating 
activities of sev ral different kinds, and therefore pro- 
vides means for doing so. 
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The present invention will now be described by 
way of example with refer nee to th accompanying 
drawings, in which: 

Fig. 1 is a diagrammatic view of a basic data-ret- 
rieval system of the present invention, in which 5 
the raw activity-based data are broken down into 
episodes which may be selectively fed to various 
output devices; 

Fig. 2 is a view of a more-complex system than 
Fig. 1 ; using two or more of the basic systems of 10 
Fig 1 feeding into a compound episode recog- : 
riser so as to be able to establish relationships 
between different forms of input data; 
Fig. 3 is a diagrammatic view of one system of the 
present invention using "active badges" to feed 15 
movement information into the system; 
Fig. 4 is a diagrammatic view of a sub-system for 
using badge episode data produced by the sys- 
tem of Fig: 3, to identify when two or more people 
form a gathering, and for forming a travelogue of 20 
the movements of people carrying badges be- 
tween different zones in the environment being 
monitored; 

Fig. 5 shows in (a) a diary of individual events in 
the life of one person, and in (b) these events sim- 25 
plified by amalgamating short periods alone into 
attendance at a gathering; 

Fig. 6 is a diagrammatic view of part of a system 
of the present invention in which the activity being 
monitored is the presence of various documents 30 
on the user's workspace, with at least descriptors 
of those documents being stored to complement 
the other data assisting in later retrieval; 
Figs. 7(a) to (c) are partial displays of a typical 
output of the Fig. 6 system; 35 
Fig. 8 is a diagrammatic view of a subsystem 
based on that system shown in Fig. 6, to break 
down the presence of documents on the work- 
space, and the timing of the appearance on, and 
disappearance from, the workspace, to produce 40 > 
document-movement episodes; 
Fig. 9 is a diagrammatic view of a sub-system of 
the present invention in which the sounds 
received and emitted by a person wearing a 
microphone are subjected to a filtering action to 45 
determine when the person carrying the micro- 
phone is talking, from when he is not By the use 
of a compound episode recog niser as shown in 
Fig. 2, conversations between two or more people 
can be recognised as such and can be used to so 
form episodes for retrieval purposes; 
Figs. 10(a) and (b) are views of typical outputs 
from the Fig. 9 system, showing how convers- 
ations are distinguished from concurrent talk; 
Fig. 1 1 is a diagrammatic view of a sub-system 55 
rather similar to that of Fig. 8, in which the input 
is of telephone calls made or rec iv d by an indi- 
vidual in th environment being monitored. By 



means of signals derived from incoming calls, or 
from utterances of the pers n being monitored, 
such phone calls can be formed into episodes, 
with any resultant diary including the names of the 
people with whom the person conversed over the 
telephone, irrespective of whether or not that per- 
son identified himself by name, or was identified 
by name by the person being monitored; and 
Fig. 12 is a diagrammatic view of a sub-system in 
' ' which the activity being monitored is that of the 
use of the electronic workstation of the person 
being monitored. 

In the drawings, many parts are specifically label- 
led as well as being given reference numbers. When 
components or units are common to two or more of the 
Figures, they retain their original references. 

The data-retrieval system of this invention is 
intended to operate on activity-based data from a 
choice of sources. 

As shown in Fig. 1 , the data may be fed directly 
to a memory 4, prior to processing by the episode rec- 
og niser 6, or the recog niser might operate on raw data 
and store the processed data in a memory 8. Because 
the information to be retrieved is not known at the time 
of data capture, it is necessary to be able to specify 
the appropriate characteristics of the data to be ret- 
rieved by means of a query station 10, having a 
keyboard 12 which the user can use to alter the par- . 
ameters of the recogniser 6 so that the raw data are 
processed into episodes having predetermined 
characteristics. 

The output of the recogniser 6 is fed into a diary 
composer 14 that is adapted to process the episode 
data fed to it and produce output data in the form of 
diary information which is fed to a driver 16. Although 
the driver is shown as operating a printer 18, a visual, 
display unit 20 and a loudspeaker or other audio 
device 22, only one or none of these devices may be 
used in any specific embodiment, but there should ; 
always be at least one output device able to present 
the data or information to the user, and to enable him 
to identify a portion of the data which he associates 
with a particular episode he is trying to recall, so that 
the time-stamp of that episode then can be used to 
access video or audio recordings of the environment ■• 
in which he was interested at the time of the episode 
in question. 

Although no specific time-keeping device is 
shown in the Fig. 1 embodiment, it is to be understood 
that all forms of apparatus according to this invention 
have to be able to operate on only time-stamped data. 
As most microprocessors which form components of 
the system of the present invention have built-in time 
clocks, wh re by the microprocessor output is time- 
stamped, even if th us r is not always able to se th 
time-stamp on first viewing, but p maps nly n 
d mand, the drawings have omitted th clock input 

That apparatus shown in Fig. 2 uses two different 
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sources 2 of raw of input data with two appropriately 
configured pisode recognisers 6. Each episode rec- : 
ogniser feeds its output to a compound episode rec- 
ogniser 24, of which the episode-recognition 
parameters are set by a query device 10, with the. 
queries being passed as necessary upstream to each 
pisode recogniser 6 feeding its output to the com- 
pound recogniser 24. This involvement of the indivi- 
dual recognisers with the compound recogniser is 
necessary in order to enable the latter to sort data into 
compound episodes involving data of two different 
types combined in a way specified by the query sta- 
tion 10. In a manner similar to the sub-system shown 
in Figure 1 , the compound recogniser 24 feeds its out- 
put to diary composer 14 and an output.device driver 
1 6 connected to selected output devices (not shown). 

As will be appreciated, the diary composer 3 is 
application-dependent, whereas the episode recog- 
niser 2 is data-dependent The compound recogniser 
accepts both simple queries, as in the Fig. 1 system, 
as well as compound queries specifying relationships 
between episodes from different sources. 

The system shown in Fig. 3 is based on the use 
of encoded identifiers, each intended to be carried by 
people working in the environment being monitored. 
One particularly-convenient form of identifier is the 
"active badge" offered by Olivetti . This identifier takes 
the form of miniaturised circuitry in a housing ab|e to 
be pinned or clipped to the clothing of the person to. 
whom the identifier is issued. The circuitry is designed 
to emit pulse width modulated infra-red coded signals 
for a tenth of a second every 12 to 15. seconds. The 
signals, by their nature; have a range of about six met- 
res, and will not travel through walls, which makes 
them very useful for keeping track of people moving ,< 
between rooms in a normal working environment In 
each of the rooms or corridors of interest (which would 
normally exclude toilets, lifts and like utility spaces) ; 
would be positioned one or more sensors responsive 
to the infra-red output signals. The sensors would be 
connected to a master station processor, which would 
have the task of polling the sensors for identifi- 
cation 'sightings'; processing the data (which would : 
include decoding the signals to determine which per- 
son was within range of a specific sensor at the time 
of polling), and presenting the processed data in a 
visual format. 

The badge system shown in Fig. 3 uses a network 
of sensors 26 of which there is at least one in each 
zone of the building or other environment being moni- 
tored. In large rooms, the effective range of the 
badges might necessitate there being several sen- 
sors uniformly distributed over the room, as by being 
mounted from th ceiling, to ensure that each occup- 
ant who is carrying a badge is in range of at I ast one 
sensor, so that the room has no 'blind spots'. The sen- 
sors 26 ar connected to a polling device 28 which 
sends pulses to the sensors at chosen intervals, and 
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receives from the sensors ncoded signals indicating 
the badges within range of each sensor at the time of 
polling. The poller 28 supplies this badge information 
to a badge server 30 having its own data memory 32. 
The badge server 30 supplies its signals to a badge 
database 36, having a data memory 38, so that the 
badge signals are decoded and the identity of the per- 
son allocated a badge becomes part of the badge sig- 
nal. The badge server 30 accepts the poller data and 
maps sensor signals into people and locations using 
its own database 32, and maps the detection of a par- 
ticular person in a particular location at a specific time 
into arrival and departure events. The database 36 
stores badge event data if appropriate. A badge 
episode recogniser 40 operates on the output from 
the database 36 to characterise the badge signals into 
three types, of episodes: gatherings; periods alone, 
and travel between or away from monitored locations. 
Similarly to what has been already described, the rec- 
ogniser 40 has its output signals treated by diary com- 
poser 14 and driver 16. 

The sub-system of Fig. 4 details the way in which 
the badge event signals may be processed. The first 
stage of the badge episode recogniser 40 of Fig. 3 is 
a gathering recogniser 42, which receives badge 
event signals, and uses an event to keep a record of 
who is at each location, and notes the formation and 
dispersion of "gatherings". This term is a convenient 
one to use, as signifying an episode worthy of being 
recorded. A working definition of a "gathering" is that 
it is regarded as an - occasion which has started when 
two or more people are present concurrently at a 
specified location, for more than a specified minimum 
period, and at least one of the participants is not resi- 
dent at that location -. This implies that the location 
database forming part of the environment identifies 
which is the office of each badge holder, so that his 
presence in his office is recognisably different from 
the presence of someone else in his office. The 
gatherings data are stored for use by a travel organ- 
iser 44, which receives badge events and supplies 
signals to a stopover list 46; the stopovers including 
locations where more than a minimum period was 
spent locations at the terminus of a round trip of more 
than a minimum distance, and the locations of gather- 
ings. 

Data from a floor-plan store 48 are used to recog- 
nise "round trips", by which term is meant - when a 
person being monitored leaves his office and moves 
around the environment before returning to it The 
stopover list 46 incorporates details of gatherings that 
are fed to the diary composer 14. The diary composer 
14 operates as illustrated in Fig- 5. The raw data 
showing a small part of the movem nts of one indivi- 
dual shown in Fig. 5 (a). This relatively-raw data would 
lead to an unnecessarily-long diary or vent log. Thus 
the diary composer 14 operates on this information to 
combin sequences of cons cutive episod s that 
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match specific patterns. Thus, for the activiti s of th 
person shown in Fig. 5 (a), the periods when he is 
al ne in the conference room are treated as being part 
of the general gathering in that room, thus leading to 
th simplification shown in Fig. 5 (b). * 

Fig. 6 shows the system which exploits scanning 
and image-processing technology to read the con- 
tents of documents on the desk surface: the user is 
th n able to interact with these contents using a 
pointer such as a tablet and stylus. A wide range of 
interaction techniques may be used, including menu 
selection and dialogue boxes, graphical selection and 
hand printed character recognition. Moreover, the 
user is also free simply to write ondocuments with a 
pen or with an ink pen in the stylus. Particular tasks 
for which the system could be effective include: 
selecting key words for document filing; performing 
calculations on numeric data; translating foreign 
words, looking up technical terms in dictionaries, 
checking and filling in forms, retrieving other docu- 
m nts or information that cannot be printed, e.g. audio 
or video. The invention could therefore give desk 
workers and people attending meetings access to 
functions that are currently available only via work sta- 
tions. The apparatus shown in Fig. 6 includes a video 
camera 50 and a video projector 52 mounted above 
a desk 54 at Which the user sits, a data tablet 56 with 
a stylus 58, a high-resolution scanner 60, and a con- 
trol computer 62 with its data store or memory 64 for 
images/ 

The camera 50 is mounted so that it field of view 
includes the document on the desk 54 in front of the * 
user. The video signal is digitised using a frame store, 
creating a bit map image of the document: Digital fil- 
tering techniques are applied to the image, first to 
determine size and rotation, and then to classify the 
document according to properties such as line spac- 
ing, line-length distribution, etc. Application of these 
techniques provides (a) a compact resolution- inde- j 
pendent encoding of the images properties for use as 
a descriptor, and (b) orientation and position infor- 
mation. The data tablet 56 and stylus 58 permit the 
user to select items such as words or columns of num- 
bers. When the user makes a selection as by pressing 
down on the tablet surface, the coordinates of the 
stylus 58 are transformed, using the inverse of the 
rotation and position information, to determine the 
selection in the coordinates of the page. 

If the image scanned in from the camera 50 is of 
insufficient resolution to support interaction, the docu- 
ment can be scanned in at high resolution. This oper- 
ation is done via the sanner 60, and it could be 
performed n th ntir contents of th user's in-tray 
bef re a work session, or on an entir set of me ting 
papers b fore a meeting starts. Th some digital fil- 
tering techniques as bef or are applied to the high- 
r solution page image to c nstruct a descriptor and t 
correct for rotation. Optical character recognition 
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(OCR) is applied to extract the text of the document 
This text is stor d for later use, indexed by the docu- 
ment descriptor. The page coordinates of each text 
item are included with the stored text 
5 The memory 64 contains layout information for 

each document page entered into the system, 
indexed by descriptor. The user need not apply high- 
resolution scanning or OCR to a document page that 
is already stored. As soon as the descriptor has been 

10 generated from the video scanned data, the existence 
of a stored version can be determined. Layout infor- 
mation describing computer-generated documents 
can be written directly to the memory 64, and descrip- 
tors generated, without recourse to scanning or OCR. 

15 The descriptor of each page of a multi-page docu- 

ment, or of each single document, is non-unique, but 
is designed to ensure that descriptors are rarely dup- 
licated. Descriptors for pairs of document images may 
be compared using a correlator to determine if the two 

20 images are of the same document When the user 
selects an item with the stylus, the corrected coordi- 
nates are used to identify the text item in the stored 
version. The appropriate function (translation, index- 
ing, etc.) can then be applied to the item. The system 

25 can also supply hand-printing recognition to the stylus 
input, or record changes to the scanned image result- 
ing from conventional handwriting. - 

Feedback to the user is provided via the video 
projector 52. Examples of such feedback are shown ' 

30 in Fig. 7: Results of translations (Fig. 7a) or calcu- 
lations (Fig. 7b) would normally be shown alongside ' 
the text If space permits (and the system can deter- v 
mine this) the results could be displayed directly 
above the selected item. The video projector 52 also *~ 

35 displays menus and dialogue boxes (see Fig. 7c) for 
interactive control. ■ ■ 

Each time the user places a new document on the 
tablet or turns the pages of a multi-page document, 
the system can detect the change in the video-scan- 

40 V ned image and then recompute the image's descrip- ; 
• tor. It searches for a matching descriptor in its file of : 
documents! Thus if it finds a match, it retrieves the 
document -If, on the other hand, no match is found,. it 
signals to the user to feed the document into the scan- 

45 ner 60. ' 

This embodiment of the invention can be com- 
pared either with workstation-based interaction or 
with traditional paper-based work practice. As an 
alternative to workstation-based interaction, it has the 

so advantage that it can deal with any incoming paper 
document or print-out of an electronic document 
Compared with paper-based methods, it has the 
advantage of providing access to a number of corn- 
put r-based functions and s rvices. It has particular 

55 advantages in me ting rooms, where us of paper 
d cuments is a firmly-trenched practice. 

The "audit traB" I ft by th system provides a 
source of data about th us r*s interaction with paper 
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documents. These data can b used to construt 
pisodes in which paper documents are identified 
either by selected key words or by optically-read con- 
tents. The subsystem shown in Fig. 8 uses the data 
from the computer 62 and armory 64 of the Fig. 6 5 
embodiment The computer feeds raw document 
vents to a document episode recogniser 66, the 
events including descriptors, commands and selected 
data. The recogniser builds episodes from consecu- 
tive events applied to the same document; episode 10 
descriptions include time-stamp, descriptor and type 
of episode (including reading, editing, filing, calculat-, 
ing). The diary composer 14 takes episodes describ- 
ing individual document pages and builds episodes 
describing activity on multi-page documents and on: 15 
folders of documents. 

While badges and other location sensors are able 
to detect that people are in the same room, they can- 
not tell whether these people form a group. Fig 9 
showns a sub-system having the ability to detect con- 20 
versations between people, without necessarily 
recording their voices, so as to provide a much more 
accurate indicator of group activity. Users typically 
remember conversations better than they remember 
other participants presence so the ability to detect that 25 
the conversation took place is likely to be useful for 
retrieval. In the Fig. 9 system, a microphone 68 is car- 
ried by a user, and the level of the audio signal pro- 
duced by the microphone is monitoredby means of a , 
circuit of the type used in cassette recorders. The cir- 30 
cuit controller 70 transmits a signal whenever the ,} 
energy level of the audio signal exceeds a threshold 
representing the wearer's minimum voice level, v 
These signals, and their time of receipt, are collected 
by the receiver 72 and passed to a conversation rec-, 35 
ogniser 74. r . 

All of the energy data sets from a particular loca- 
tion are analysed together. The analysis seeks to 
identify periods of. high energy as utterances; and to y A 
fit together utterances of two or more users so as to ?- 40 
form continuous conversations. The methods draws 
on social science research results showing that con-, 
versation involves relatively-little overlap and relative- 
ly-short pauses between turns. Thus the composite 
audio stream is more or less continuous, with mostly 45 
only one speaker active. Fig. 10 (a) shows the audios- 
treams generated by participants in a typical convers- 
ation, while Fig. 10 (b) shows, in contrast, the 
audiostreams of two people engaged in convers- 
ations with others or in dictating, etc.. Each composite 50 
signal can be divided into periods of silence, non- 
overlapping speech, and, overlapping speech. In true 
conversation between two people, overlap would tend 
to r present less than 1 0 percent of all non-sil nee, 
while in random collections of speakers, the propor- 55 
tion will be considerably higher, at least 25 perc nt. 

Although this has not been shown in th draw- 
ings, an alternativ method of capturing th conv rs- 



ation data is by radio telephony. Audio signals from 
us rs' microphones are collected at the central loca- 
tion, their energy levels are measured, and the con- 
versation-detection analysis carried out This is a 
more-expensive approach that offers increased 
bandwidth, but it may encounter channel limitations. 

The diary composer 14 reduces the list of 
episodes by excluding short episodes, and by linking 
conversations between identical groups. 

Fig. 11 shows a sub-system that builds a log of 
incoming and outgoing telephone calls that includes 
the names of respondents. The proposed device 
could be attached to a user's telephone and would be 
capable of building a text database, each record indi- 
cating the time and duration of a call, and both the cal- 
ler's and the recipients names, with one of the names 
being generated by a speech-recognition system. The 
system shown in Fig. 11 uses input from a telephone 
76 from which signals are picked off and fed to a 
speech recogniser 78 that has its output routed to a 
telephone call episode recogniser 80, with its output, 
in turn, going to the now-usual composer 14. Output 
from the composer 14 builds the log, of which a typical 
set of entries is shown at 84. 

This aspect of the invention relies on recognising 
names spoken by the user in responding to incoming 
calls and in making outgoing calls. 

On making outgoing calls the speech recogniser 
78, which is monitoring the audio signal continuously, 
will recognise an utterance of the form "Patrick 
Johnson please" or "Could I speak to Patrick", etc. As 
long as the person being called has corresponding 
voice identification data entered in a database 82, the 
utterance by the caller of the spoken name produces 
the audio signals which are fed to recogniser 80 f The 
recogniser 80 generates the recipient's name as a 
text string. On receiving incoming calls, the speech 
recogniser 80 further responds to a recipient's utter- ; 
ance of the form "Hello Patrick" or "Hello Mr Johnson" 
and generates the caller's name as a text string.This 
recognisergenerated data is stored, so it may be used 
for subsequent manual confirmation by the user. 

The diary composer 14 may combine consecutive 
calls to the same person, and ignore wrong numbers 
and interception by switchboards, etc. If the caller is 
not known to the user, or if the user calls somebody 
whose data is not in database 82, the system cannot 
supply the identification data. There is also the pos- 
sibility that, in the case of an incoming caJI, the reci- 
pient may not respond, such as when speaking to 
callers who are relatively strange to him, in the way for 
which the recogniser 80 is programmed. This vari- 
ation in response, or indeed any utterance in which 
the confid nee of r cognition is low, can be used to 
mark the call r as unknown in the log ntry. 

The sub-system shown in Fig. 12 uses the activity 
of a user's workstation to generate episod data. A 
workstation activity epis de recogniser 86 has a data 
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memory 88. The recognis r has various forms of input 
from a workstation, as shown in the drawing. One 
input might relate to file activity, another to processor 
focus, and a third to program input focus. These 
inputs , are from daemons or watchdog processes. 5 
These monitor activity such as: the creation, reading, 
modifying or deleting of files; the name of a program 
c nsumihg processor resource in a given period, or 
changes in program input focus (in a window-based 
user interface). The recogniser 86 translates sequ- 10 

nces of activity into user-recognisable episodes. 
These include: editing a document; reading, answer- 
ing and filing mail; developing a program; and retriev- 
ing information from a database. The recogniser 86 
applies pattern-matching to recognise common 15 
tasks . i.e. "change input focus to editor ^ change 
processor focus to editor read file write file -> 
write file — ► change input focus", would be recognised 
as being the steps involved in editing a document The 
composer 14 combines similar episodes and removes 20 
insignificant episodes. ■ 

Users can submit queries to each of the recognis- 
ers described above by means of its respective query 
input 10. A query will specify an episode or episodes 
of interest, including a pattern of episodes, using key 25 
words and descriptions appropriate to the recogniser, 
for example: 

- with KBW in Commons last week; ; : ' 

- document read after reading Mid-year Report; 

- talking to KBW and MGL; 30 

- calls to MGL yesterday; and 

- reading mail yesterday: = 

In the case of the compound recogniser 24 (Fig 
2) such a query might involve a pattern of the form: 

- sending mail to KBW after meeting with MGL. 35 
Although the different embodiments of the inven- 
tion are implemented in the form of computer code, 
which is not included in this specification, it is never- 
theless believed that the above description and draw- 
ings are sufficient to enable a man skilled in the art of 40 
designing software systems to put this invention into 
practice. 



arriv at records of episodes matching a specified 
search pattern, in which each episode record is 
made up of periods spent alone by the person 
concerned; in travel within the environment, and 
in attendance at gatherings, and 

means for requesting transformation into 
human-readable form of the episode records cor- 
responding to the specified pattern. 

2. A system as claimed in claim 1 , in which the envi- 
ronment is divided into several zones, and includ- 
ing: 

means for monitoring one or more of the 
zones to detect the presence within the zone of. 
and to identify, a person. 

3. A system as claimed in claim 1 or 2, in which at 
least some of the people in the environment carry 
a personal identifier from which data can be col- 
lected by one or more monitors of the environ- 
ment 

4. A system as claimed in claim 3, in which each 
identifier is in the form of a transmitter of identifi- * 
cation signals to which each monitor is sensitive. 

5. A system for providing for fast random access to 
a sequence of time-stamped data records of 
events in a specific environment, comprising: 

means for collecting data in order to con- 
struct time-stamped records identifying scann- - 
able document pages and indicating operations " 
on the pages at locations in the environment; 

means for encoding and storing the-t 
• records for later selective retrieval and decoding; 

means for enabling identification of the 
document contents from a record. 

6. A system as claimed in claim 5, including means 
Tfor projecting a display on to the surface of the 

document or on to a surface bordering the docu- 
ment :•■ 



Claims 



45 



A system for providing for fast random access to 
a sequence of time-stamped data records of 
events in a specific environment, comprising: 

means for collecting data in order to con- so 
struct time-stamped records indicating the loca- 
tions of persons in the environment; 

means for ncoding and storing th 
records for later s I ctiver trieval and decoding; 

m ans for specifying a pattern characteri- 55 
sing an episod as d fined, and for receiving and 
d coding matching records; 

means for applying rules to the records to 



7. A system as claimed in claim 5 or 6. including 
means for manually selecting from within this 
document or for displaying a computer function to 
be applied to the document. 

8. A system as claimed in any of claims 5 to 7, 
including means for manually selecting one or 
more items on the identified document to be used 
as input to a computer function to be applied to 
the document 

9. A system for providing for fast random access to 
asequ nee of time-stamped data r cords of 
activities in a sp crfic environm nt , comprising: 

m ans for collecting data in ord r to con- 
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struct time-stamped r cords of periods of vocal 
sounds by people in the environment; 

means for operating on those records to 
determine when a conversation is taking place 
involving one or more of the said people, and for 5 
generating time-stamped records of such deter- 
minations; 

means for encoding and storing the 
records for later selective retrieval and decoding, 
and 10 

means for enabling identification of the col- 
lected conversation data from a record. 

10. A system for providing for fast random access to 

a sequence of time-stamped data records of 15 
activities in a specific environment, comprising: 

means for collecting audio signal and other 
data from a person in the environment placing or 
receiving a phone call in order to construct time- 
stamped records of calls; 20 

means for feeding audio signals from 
these records to a speech recogniser capable of 
identifying the other participant in the call from the 
words uttered by the person in the environment, 
and for adding this identification to the records; 25 

means for encoding and storing the 
records for later selective retrieval and decoding, 
and i 

means for enabling identification of thecal! . 
data from a record. 3d 



records of what was said in each of the zones. 

14. A system as claimed in any preceding claim, cap- 
able of generating episodes of two or more types, 
including means for arriving at episode records 
satisfying a - specified relationship between 
episodes of two or more types. 

15. A system as claimed in any preceding claim, 
including: 

means for specifying a pattern characteri- 
sing an episode as defined, and for receiving and 
decoding matching records; 

means for applying rules to the records to 
arrive at records of episodes matching a specified 
search pattern, and 

means for requesting transformation into 
human-readable form of the episode records cor- 
responding to the specified pattern. 



1 1. A system for providing for fast random access to 
a sequence of time-stamped data records of 
activities in a specific environment, comprising: 

means for collecting data in order to con- 35 
struct time-stamped records of the activity of each 
workstation associated with each person in the 
environment, such as the creation, reading, modi- 
fication or deletion of electronic files; 

means for encoding and storing the 40 
records for later selective retrieval and decoding; 

means for specifying a pattern characteri- 
sing an episode, and for receiving and decoding 
matching records, and 

means for e nabl i ng identification of the col- 45 
lected data from a record. 



12. A system as claimed in claims 4 arid 9, in which 
each transmitter has associated with it a micro- 
phone for monitoring the level of the audio signals 50 
produced by the person carrying the transmitter, 

the microphone circuit having a filter for rejecting 
all signals of less than a threshold level, and 
transmitting data describing p riods of vocal 
sounds greater than the threshold I vel. 55 

1 3. A system as claimed in any preceding claim, com- 
prising means for producing at I ast auditory 
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(g) Indexing of data sets. 



(g) A system is disclosed for facilitating later 
access to a portion of a datastream for retrieval 
purposes based upon identifying "episodes" in 
data from other sources related to a user's 
activities. The stored data, in raw or "episodic " 
form, are accessed as desired using parameters 
based on recollections of the user's or database 
interrogator's activities at the time of the event 
which the user or interrogator wishes to access 
or display. Such other sources may include 
information about the movements, gatherings, 
conversations, phone calls, etc. of users in a 
particular environment fitted with the system. 



CO 

< 

CM 
CM 
CO 

in 



Q. 
UJ 



26 ^ 26 

r-o — ci — I 



Fig. 3. 

■a □ 



BADGE 
DATABASE 



28 



1*0 



BADGE 

EPISODE 

RECOGNISES 



QUEPY-^ = 



14' 



Jouve, 18, rue Saint-D nis, 75001 PARIS 



ISDOCia <EP 0495622A3_I_> 



v V 



EP 0 495 622 A3 



European Patent 



EUROPEAN SEARCH REPORT 



EP 92 30 0302 
Page 1 



DOCUMENTS CONSIDERED TO BE RELEVANT 



Category 



Citation eft 



off relevant passages 



US-A-6 479 631 (DINNAT R.) 

* the whole document * 

PATENT ABSTRACTS OF JAPAN ■ r 

vol. 010, no. 256 (P-493)2 September 1986 
& JP-A-61 082 287 ( TOSHIBA ENG CO LTD ) 
25 April 1986 

* abstract * 

US-A-4 468 694 (EDGAR A.) 

* the whole document * 

EP-A-Q 175 503 (BRITISH TELECOMMUNICATIONS 
PLC) 

* the whole document * 

PATENT ABSTRACTS OF JAPAN 
vol. 012, no. 293 (E-645)10 August 1988 
& JP-A-63 067 949 ( FUJITSU LTD ) 26 March 
1988 

abstract * 

PC TECH JOURNAL . 
vol. 6, no. 2, February 1988, USA 
pages 126 - 134 \ ., B 

SAWICKI E. •The LAN Audit Trail 1 

IEEE PROCEEDINGS OF THE 6TH ANNUAL 
COMPUTER SECURITY APPLICATIONS CONFERENCE, 
CAT. NO. 90TH0251-7, 7 December 1990, 
TUCSON, AZ, USAi 
pages 260 - 272 

MCAULIFFE ET AL. 'Is Your Computer Being 
Misused ? A Survey of Current Intrusion 
Detection System Technology 1 

-/~ 



1,2 
3,4 



3,4 



5-8 



Rcfavaot 

to< 



9,10,12, 
13 



9,10,12, 
13 



11,14,15 



11,14,15 



CLASSIFICATION OF THE 
APPLICATION (fart. CL5 ) 



G06F15/401 
G06F15/403 



TECHNICAL FIELDS 
SEARCHED <tot CL5 ) 



G06F 



The present seven report has been <fc»wn np for all da 



THE HAGUE 



06 AUGUST 1993 



SUENDERMANN R.O. 



CATEGORY OF CITED DOCUMENTS 



T : theory or principle uneertymt 
WL : emitter oaten* document, out p 



X : particularly relevant if 
Y : putkubsty relevant if 
fecument of the 

A: 
O: 
P: 



wife 



after the filing *ste 
D : documeat cite* to the 
L 



lofthes 



fanUK/, 



MSDOCID: <EP 0485622A3_I_> 



EP 0 495 622 A3 



European Patent 
Office 



EUROPEAN SEARCH REPORT 



EP 92 30 0302 
Page 2 



DOCUMENTS CONSIDERED TO BE RELEVANT 



Category 



Citation of 



opnst 



CLASSIFICATION OF THE 
APPLICATION Oat> CLS ) 



GB-A-2 142 500 (UNITED KINGDOM ATOMIC 
ENERGY AUTHORITY) 

* the whole document * 

US-A-4 884 132 (MORRIS ET AL.) 

* the whole document * 



1,5,9 



TECHNICAL FIELDS 
SEARCHED (Int. CLS ) 



Tbe present 



drawn up for at] i 



THE HAGUE 



D**«f oaaaMamatflaj ma 

06 AUGUST 1993 



SUENOERMANN R.0. 



s 



i f 



CATEGORY OF CITED DOCUMENTS 

X : pertkadarty relevant H takes atone 
Y : psrtlcalaiiy relevant If combined with 
oacpjneai of tbc tv 



T : taenry or principle 
E : cutler patent * 



g tba invuittoej 



attar tha Mag date 
D : cocaDMDt cite* in tha applkatiop 
L:aocanaBt< 



P: inumaitate 



r of thai 



SSDCC1D: < E P 0495622A3_I_> 



THIS PAGE BLANK (USPTO) 




